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(57) An image processing apparatus includes: an im- 
age acquisition section which acquires an original image; 
a resolution conversion section which converts the res- 
olution of the original image acquired by the image ac- 
quisition section and generates a plurality of reduced im- 
ages having different resolutions; a detection section 
which processes by template matching the plurality of 
reduced images generated by the resolution conversion 
section and detects an area occupied by a picked-up 
image of a particular object corresponding to the tem- 
plate, from the reduced images; and a detection result 
processing section which detects the area occupied by 
the picked-up image of the particular object on the original 
image, by processing a detection result obtained by the 
detection section, the detection section detecting the ar- 
ea occupied by the picked-up image of the particular ob- 
ject by processing the plurality of reduced images in an 
order in which resolution sequentially varies on a step- 
by-step basis. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present invention relates to an image 
processing apparatus, an image processing method, a 
program for the image processing method, and a record- 
ing medium which records the program for image 
processing method, and can be applied, for example, to 
digital still cameras. The present invention makes it pos- 
sible to detect areas respectively occupied by picked-up 
images of particular objects by performing template 
matching of reduced images in an order in which reso- 
lution sequentially varies on a step-by-step basis, thereby 
detecting the respective areas occupied by the picked- 
up images of the particular objects and appropriately set- 
ting the priority order. 

2. Description of Related Art 

[0002] In the field of digital still cameras, video camer- 
as and the like, it has heretofore been proposed to pro- 
vide a method of detecting an area occupied by a picked- 
up image of a particular object from an image pickup 
result and controlling an image pickup system on the ba- 
sis of the image pickup result of the area. In this method, 
the face of a person is mainly applied to the particular 
object, and an area occupied by a picked-up image of 
the particular object, for example, an area of skin color, 
is detected as by template matching using a template. 
[0003] As to such a method, Japanese Patent Appli- 
cation Publication Number 2004-30629 proposes a de- 
vice related to detection of an area occupied by a picked- 
up image of a face in template matching using a template. 
[0004] The processing of detecting an area occupied 
by a picked-up image of a particular object from an image 
pickup result in this manner needs to be executed at suf- 
ficiently high speed so that the process can track the 
movement of an image pickup apparatus and an object. 
In addition, there is a case where areas respectively oc- 
cupied by picked-up images of particular objects are de- 
tected at a plurality of locations, and in this case, it is 
necessary to appropriately set the priority order of the 
areas detected at the plurality of locations in order to 
determine which of the areas is to be processed at high- 
est priority. 

SUMMARY OF THE INVENTION 

[0005] The present invention has been made in view 
of the above-mentioned issue, and provides an image 
processing apparatus and method capable of detecting 
an area occupied by a picked-up image of a particular 
object and appropriately setting the priority order, and a 
program for the image processing method, as well as a 
recording medium which records the program for the im- 



age processing method. 

[0006] In accordance with a first preferred embodiment 
of the present invention, there is provided an image 
processing apparatus which includes an image acquisi- 

5 tion section which acquires an original image, a resolu- 
tion conversion section which converts the resolution of 
the original image acquired by the image acquisition sec- 
tion and generates a plurality of reduced images having 
different resolutions, a detection section which process- 

10 es by template matching using a template the plurality of 
reduced images generated by the resolution conversion 
section and detects an area occupied by a picked-up 
image of a particular object corresponding to the tem- 
plate, from the reduced images, and a detection result 

15 processing section which detects the area occupied by 
the picked-up image of the particularobjecton theoriginal 
image, by processing a detection result obtained by the 
detection section, the detection section detecting the ar- 
ea occupied by the picked-up image of the particular ob- 

20 ject by processing the plurality of reduced images in an 
order in which resolution sequentially varies on a step- 
by-step basis. 

[0007] In accordance with a second preferred embod- 
imentof the present invention, there is provided an image 
25 processing method which includes an image acquisition 
step of acquiring an original image, a resolution conver- 
sion step of converting the resolution of theoriginal image 
acquired in the image acquisition step and generating a 
plurality of reduced images having different resolutions, 
30 a detection step of processing by template matching us- 
ing a template the plurality of reduced images generated 
in the resolution conversion step and detecting an area 
occupied by a picked-up image of a particular object cor- 
responding to the template, from the reduced images, 
35 and a detection result processing step of detecting the 
area occupied by the picked-up image of the particular 
object on the original image, by processing a detection 
result obtained in the detection step, the detection step 
detecting the area occupied by the picked-up image of 
40 the particular object by processing the plurality of re- 
duced images in an order in which resolution sequentially 
varies on a step-by-step basis. 

[0008] In accordance with a third preferred embodi- 
ment of the present invention, there is provided a program 
45 for an image processing method, which processes im- 
ages by being executed by operation processing means, 
the program including an image acquisition step of ac- 
quiring an original image, a resolution conversion step 
of converting the resolution of theoriginal image acquired 
50 in the image acquisition step and generating a plurality 
of reduced images having different resolutions, a detec- 
tion step of processing by template matching using a tem- 
plate the plurality of reduced images generated in the 
resolution conversion step and detecting an area occu- 
55 pied by a picked-up image of a particular object corre- 
sponding to the template, from the reduced images, and 
a detection result processing step of detecting the area 
occupied by the picked-up image of the particular object 
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on the original image, by processing a detection result 
obtained in the detection step, the detection step detect- 
ing the area occupied by the picked-up image of the par- 
ticular object by processing the plurality of reduced im- 
ages in an order in which resolution sequentially varies 
on a step-by-step basis. 

[0009] In accordance with a fourth preferred embodi- 
ment of the present invention, there is provided a record- 
ing medium which records a program for an image 
processing method of processing images by being exe- 
cuted by operation processing means, the program in- 
cluding an image acquisition step of acquiring an original 
image, a resolution conversion step of converting the res- 
olution of the original image acquired in the image acqui- 
sition step and generating a plurality of reduced images 
having different resolutions, a detection step of process- 
ing by template matching using a template the plurality 
of reduced images generated in the resolution conver- 
sion step and detecting an area occupied by a picked- 
up image of a particular object corresponding to the tem- 
plate, from the reduced images, and a detection result 
processing step of detecting the area occupied by the 
picked-up image of the particular object on the original 
image, by processing a detection result obtained in the 
detection step, the detection step detecting the area oc- 
cupied by the picked-up image of the particular object by 
processing the plurality of reduced images in an order in 
which resolution sequentially varies on a step-by-step 
basis. 

[0010] The image processing apparatus according to 
the first preferred embodiment of the present invention 
includes the image acquisition section which acquires an 
original image, the resolution conversion section which 
converts the resolution of the original image acquired by 
the image acquisition section and generates a plurality 
of reduced images having different resolutions, the de- 
tection section which processes by template matching 
using a template the plurality of reduced images gener- 
ated by the resolution conversion section and detects an 
area occupied by a picked-up image of a particular object 
corresponding to the template, from the reduced images, 
and the detection result processing section which detects 
the area occupied by the picked-up image of the partic- 
ular object on the original image, by processing a detec- 
tion resultobtained by the detection section, the detection 
section detecting the area occupied by the picked-up im- 
age of the particular object by processing the plurality of 
reduced images in an order in which resolution sequen- 
tially varies on a step-by-step basis. According to this 
construction, areas respectively occupied by picked-up 
images of particular objects are detected in order from 
the largest object or in order from the smallest object, so 
that the priority order related to the sizes of the areas can 
be set on the basis of the order of detection. In addition, 
the process can be stopped as needed to detect an ob- 
jective area in a short time, so that the areas occupied 
by the picked-up images of the respective particular ob- 
jects can be detected at high speed to appropriately set 



the priority order. 

[0011] In addition, according to the second, third and 
fourth preferred embodiments, it is possible to provide 
an image processing method capable of detecting areas 

5 respectively occupied by picked-up images of particular 
objects and appropriately setting the priority order, and 
a program for the image processing method, as well as 
a recording medium which records the program for the 
image processing method. 

10 [0012] According to the embodiments of the present 
invention, it is possible to detect an area occupied by a 
picked-up image of a particular object and appropriately 
set the priority order. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] The invention will become more readily appre- 
ciated and understood from the following detailed de- 
scription of preferred embodiments of the invention when 
20 taken in conjunction with the accompanying drawings, in 
which: 

Fig. 1 is a flowchart showing the process sequence 
of a central processing unit in an image pickup ap- 
25 paratus according to a first embodiment of the 
present invention; 

Fig. 2 is a block diagram showing the image pickup 
apparatus according to the first embodiment of the 
present invention; 
30 Fig. 3 is a block diagram showing a detailed con- 

struction of a face detection section in the image pick- 
up apparatus shown in Fig. 2; 
Fig. 4 is a schematic diagram aiding in explaining 
the processing of the central processing unit in the 
35 image pickup apparatus according to the first em- 
bodiment of the present invention; 
Fig. 5 is a flowchart showing a continuation of Fig. 1 ; 
Fig. 6 is a schematic diagram aiding in explaining 
the generation processing of reduced images; 
40 Fig. 7 is a schematic diagram aiding in explaining 

the processing of detection results; 
Fig. 8 is a schematic diagram aiding in explaining 
the generation processing of reduced images in the 
image pickup apparatus according to the first em- 
45 bodiment of the present invention; and 

Fig. 9 is a block diagram showing a recording and 
apparatus according to a second embodiment of the 
present invention. 

50 DESCRIPTION OFTHE PREFERRED EMBODIMENTS 

[0014] Preferred embodiments of the present inven- 
tion will be described below in detail with reference to the 
accompanying drawings. 
55 [001 5] A first embodiment of the present invention will 
be described below. 
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(1) Construction of the First Embodiment 

[001 6] Fig. 2 is a blockdiagram showing an image pick- 
up apparatus according to the first embodiment of the 
present invention. An image pickup apparatus 1 is a dig- 
ital still camera which acquires and records an image 
pickup result of a desired object in the form of a still image 
or a moving image. 

[001 7] In the image pickup apparatus 1 , a lens 2, under 
the control of a central processing unit (CPU) 3, converg- 
es incident light while varying its aperture, focus and 
zoom ratio, to form an optical image of an object on an 
image pickup surface of an image sensor 4. 
[0018] The image sensor 4 outputs an image pickup 
result of the optical image formed on the image pickup 
surface by the lens 2, in the form of a moving image or 
a still image, through the photoelectric conversion 
processing of individual photosensitive elements ar- 
ranged on the image pickup surface. 
[0019] A camera signal processing section 6, under 
the control of the central processing unit 3, receives the 
image pickup result outputted from the image sensor 4, 
executes signal processing such as matrix operation, 
gamma correction and white balance adjustment, and 
outputs image data representative of the result of the 
signal processing to an image bus BUS. 
[0020] In addition, during this signal processing, the 
camera signal processing section 6 generates and out- 
puts image data for use in monitoring in an image display 
section 9 and original image data for use in face detection 
in a face detection section 8. In the first embodiment, the 
original image data uses the monitoring image data, for 
example, image data conforming to a VGA (Video Graph- 
ics Array) format of 640 pixels x 380 pixels or image data 
of 320 pixels x 240 pixels. Accordingly, in the first em- 
bodiment, the lens 2, the image sensor 4 and the camera 
signal processing section 6 constitute an image acquisi- 
tion section which acquires image data related to an orig- 
inal image representative of an image pickup result in 
the form of a moving image or a still image. 
[0021] An image RAM (Random Access Memory) 7, 
under the control of the central processing unit 3, tem- 
porarily stores the image data outputted to the image bus 
BUS and outputs the stored image data to the image bus 
BUS. 

[0022] A face detection section 8, under the control of 
the central processing unit 3, acquires the original image 
data recorded on the image RAM 7 and detects an area 
occupied by a picked-up image of a particular object from 
the image represented by the original image data. In the 
first embodiment, this particular object is set to the face 
of a person, and the face detection section 8 notifies the 
central processing unit 3 of a detection result D 1 . 
[0023] Specifically, as shown in Fig. 3, in the face de- 
tection section 8, a resolution conversion section 8A per- 
forms filtering to convert the resolution of the image data 
stored in the image RAM 7 according to a scale factor a 
designated by a controller 8B, thereby reducing the size 



of the image represented by the image data stored in the 
image RAM 7 according to the scale factor a and output- 
ting the image of the reduced size. 
[0024] An image memory 8C, under the control of the 
5 controller 8B, records and holds the image data outputted 
from the resolution conversion section 8A and stores the 
held image data into the image RAM 7 and a face detec- 
tion core 8D. In this manner, the face detection section 
8 reduces the size of the original image stored in the 
10 image RAM 7 according to any of various scale factors 
and stores an image of reduced size in the image RAM 
7, and further reduces the size of the image stored in the 
image RAM 7 according to various scale factors and 
stores images of various reduced sizes, thereby gener- 
ating reduced images in each of which the resolution of 
the original image stored in the image RAM 7 is reduced 
to a different extent. 

[0025] The face detection core 8D detects an area oc- 
cupied by a picked-up image of the face from an image 
represented by the image data outputted from the image 
memory 8C, by template matching using a template un- 
der the control of the controller 8B. Namely, the face de- 
tection core 8D temporarily records and holds the image 
data outputted from the image memory 8C, and sequen- 
tially selects the held image data and executes correla- 
tion value detection processing on the selected image 
data as well as image data which constitutes a template, 
thereby scanning the template on the image represented 
by the selected image data and detecting correlation val- 
ues indicative of the extent of similarity between the target 
image and the template at individual scanning positions. 
In the first embodiment, the detection of each of the cor- 
relation values is executed by calculating the sum of ab- 
solute differences of luminance levels between overlap- 
ping pixels, but instead of this processing, it is possible 
to use various other techniques such as performing log- 
ical operation by representing each of a target image and 
a template in binary format. 

[0026] The face detection core 8D makes a decision 
as to the correlation values detected in this manner on 
the basis of a predetermined threshold, thereby detecting 
the area occupied by the picked-up image of the face. 
The face detection core 8D holds in its memory a plurality 
of kinds of templates each of which is set to the same 
sampling rate in the horizontal and vertical directions and 
which respectively correspond to different faces, for ex- 
ample, an image of a face taken from the front, an image 
of a face taken obliquely from the front, and an image of 
a round face. In addition, the face detection core 8D ex- 
ecutes a sequence of processing for detection of corre- 
lation values in a simultaneous parallel manner by using 
such plurality of kinds of templates each having the same 
size, thereby reliably detecting an area occupied by a 
picked-up image of a face irrespective of different ob- 
jects. 

[0027] In addition, the face detection core 8D, in ac- 
cordance with an instruction from the controller 8B, ex- 
ecutes the correlation value detection processing while 
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correcting the position of the target image with respect 
to the plurality of kinds of templates so as to correct the 
position and the inclination of the image pickup apparatus 
1 according to the position thereof detected by a gravity 
direction sensor 1 3 which will be described later, thereby 
reliably detecting an area occupied by a picked-up image 
of a face, even when a user changes the position of the 
image pickup apparatus 1 to take an image with a vertical 
angle of view, for example. 

[0028] In addition, the face detection core 8D, in ac- 
cordance with an instruction from the controller 8B, 
switches the start position of scanning, the order of scan- 
ning, and the end position of scanning, thereby detecting 
an area occupied by a picked-up image of a face in a 
short time with sufficient accuracy. Specifically, in many 
cases, if an image-taking mode in the image pickup ap- 
paratus 1 is set to, for example, a portrait mode which is 
an image-taking mode for taking images of persons, a 
picked-up image of the face of a person is located in the 
central section of the screen. Accordingly, in this case, 
the face detection core 8D, under the control of the con- 
troller 8B, starts scanning at the center of the screen and 
causes a template to scan helically toward the periphery 
of the screen, thereby detecting an area occupied by the 
picked-up image of the face of the person. In addition, 
the face detection core 8D stops scanning at the outer- 
most periphery to reduce the time required to detect the 
face, thereby detecting the area occupied by the picked- 
up image of the face with practically sufficient accuracy. 
Conversely, in the case of a group photograph, it is ex- 
pected that the faces of individual persons are detected 
at different locations of the screen. Accordingly, in this 
case, the face detection core 8D detects all areas occu- 
pied by picked-up images of the respective faces, by per- 
forming scanning in the order of raster scanning, for ex- 
ample. 

[0029] In this manner, the face detection core 8D no- 
tifies the central processing unit 3 of the detection result 
D1 of face detection, i.e., the position of the areaoccupied 
by the picked-up image of the face that has been detected 
by making a decision as to the correlation values, along 
with the size of a template subjected to the detection of 
the face. 

[0030] The controller 8B controls the respective oper- 
ations of the resolution conversion section 8A, the image 
memory 8C and the face detection core 8D under the 
control of the central processing unit 3. 
[0031] The image display section 9 includes, for ex- 
ample, a liquid crystal display device and the peripheral 
section thereof, and displays an image represented by 
the monitoring image data recorded on the image RAM 
7, under the control of the central processing unit 3. In 
addition, at this time, the image display section 9, in ac- 
cordance with an instruction from the central processing 
unit 3, displays a frame having a rectangular shape to 
surround the face, on the basis of the detection result 
from the face detection section 8. 
[0032] Accordingly, the image pickup apparatus 1 is 



constructed so that the image data inputted through the 
lens 2, the image sensor 4 and the camera signal 
processing section 6 in the form of a moving image or a 
still image can be displayed on the image display section 

5 9 for monitoring purpose so as to permit confirmation of 
the area occupied by the picked-up image of the face in 
accordance with an instruction from the user. 
[0033] An image compression/decompression section 
10, under the control of the central processing unit 3, 

10 acquires the image data recorded on the image RAM 7 
and compresses the image data by a technique such as 
JPEG (Joint Photographic Coding Experts Group) or 
MPEG (Moving Picture Experts Group), and records im- 
age data representative of the result of the processing 

15 on an image recording medium 1 2 in the form of an image 
file. As opposed to this processing, the image compres- 
sion/decompression section 1 0 decompresses an image 
file recorded on the image recording medium 12 and 
records image data representative of the result of the 

20 processing on the image RAM 7. 

[0034] Accordingly, the image pickup apparatus 1 is 
constructed so that the image pickup result acquired in 
the form of a moving image or a still image can be re- 
corded on the image recording medium 1 2 and the image 

25 data file recorded on the image recording medium 12 can 
be variously processed. 

[0035] The image recording medium 12 is any of var- 
ious recording media such as memory cards, optical 
disks, magnetic disks and magnetic tape, and records 

30 various data outputted from the image compression/de- 
compression section 10 and the central processing unit 
3 and outputs the recorded various data to each of the 
image compression/decompression section 10 and the 
central processing unit3. In addition, the image recording 

35 medium 1 2 may be either of a removable type or a built- 
in type which is difficult to remove, or both of the two 
types. 

[0036] In addition, the image compression/decom- 
pression section 10 communicates image data with ex- 

40 ternal devices via a wired or wireless data communication 
section instead of using such a recording medium. 
[0037] Accordingly, in the image pickup apparatus 1, 
the image compression/decompression section 10 con- 
stitutes an image acquisition section associated with a 

45 recording medium, for acquiring image data representa- 
tive of an original image recorded on the recording me- 
dium, and also constitutes a data communication section 
which performs data communication with external devic- 
es. 

50 [0038] Accordingly, if the user selects an image-taking 
mode for still images, the image pickup apparatus 1 se- 
quentially acquires an image pickup result from the image 
sensor 4 in the form of a still image and performs signal 
processing on the image pickup result in the camera sig- 

55 nal processing section 6, and then stores the processed 
image pickup result into the image RAM 7 and also caus- 
es the image display section 9 to display the image pickup 
result stored in the image RAM 7 in the form of a still 
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image, so that the user can monitor an image pickup 
target. In addition, during this monitoring state, if the user 
operates a trigger switch (not shown), the image pickup 
apparatus 1 acquires an image pickup result in the form 
of a still image instead of the image pickup result that 
has so far picked-up in the form of a moving image, and 
stores the acquired image pickup result into the image 
RAM 7, and also causes the image display section 9 to 
display the image pickup result stored in the image RAM 
7 in the form of a still image, so that the user can monitor 
the image pickup result. In addition, if the user instructs 
the image pickup apparatus 1 to record the image pickup 
result, the image pickup apparatus 1 compresses the im- 
age data stored in the image RAM 7 in the form of a still 
image by means of the image compression/decompres- 
sion section 1 0, and records the compressed image data 
on the image recording medium 12. 
[0039] Accordingly, if the user selects an image-taking 
mode for moving images, the image pickup apparatus 1 
sequentially acquires an image pickup result from the 
image sensor 4 in the form of a moving image and per- 
forms signal processing on the image pickup result in the 
camera signal processing section 6, and then stores the 
processed image pickup result into the image RAM 7 and 
also causes the image display section 9 to display the 
image pickup result stored in the image RAM 7 in the 
form of a moving image, so that the user can monitor an 
image pickup target in this case as well. In addition, dur- 
ing this monitoring state, if the user operates the trigger 
switch, the image pickup apparatus 1 causes the image 
compression/decompression section 10 to sequentially 
compress the image data stored in the image RAM 7 and 
records the compressed image data on the image re- 
cording medium 12. 

[0040] In addition, if the user instructs the image pickup 
apparatus 1 to reproduce an image file recorded on the 
image recording medium 12 in the form of a still image 
or a moving image, the image pickup apparatus 1 ac- 
quires image data representative of the image file from 
the image recording medium 1 2, causes the image com- 
pression/decompression section 10 to decompress the 
image data, and stores image data representative of the 
resultof the processing into the image RAM 7. In addition, 
the image pickup apparatus 1 generates monitoring im- 
age data from the image data stored in the image RAM 
7 and causes the image display section 9 to display an 
image represented by the monitoring image data. 
[0041] The gravity direction sensor 13 includes an ac- 
celeration sensor for detecting acceleration in different 
directions and a signal processing section for processing 
a detection result outputted from the acceleration sensor 
and detecting the direction of gravitational acceleration, 
and detects the position of the image pickup apparatus 
1 and notifies the central processing unit3 of the detected 
position. 

[0042] A memory 1 4 is formed by a nonvolatile memory 
and a volatile memory, and records a program for the 
central processing unit3, data required for the processing 



of the central processing unit 3, and the like, and also 
forms a work area and the like for the central processing 
unit 3. 

[0043] The central processing unit 3 is a control section 
5 for controlling the operation of the image pickup appara- 
tus 1 , and executes the program recorded on the memory 
14 to control the operations of the respective sections in 
response to operation performed by the user. In the first 
embodiment, the program is provided in the form of being 
10 preinstalled in the image pickup apparatus 1 , but may be 
provided in the form of being recorded on various record- 
ing media such as optical disks, magnetic disks or mem- 
ory cards, instead of being preinstalled, or may also be 
provided by downloading from a network such as the 
15 Internet. 

[0044] The central processing unit3 executes the proc- 
ess sequence of this program to acquire the image pickup 
result in the form of a moving image or a still image in 
response to an instruction from the user and display the 

20 acquired image pickup result on the image display sec- 
tion 9, and also records the image pickup result on the 
image recording medium 12. At this time, the central 
processing unit 3 instructs the face detection section 8 
to perform operation and acquires the detection result 

25 D1 , and acquires image data representative of the area 
occupied by the picked-up image of the face on the basis 
of the detection result D1 . On the basis of the acquired 
image data, the central processing unit 3 controls the 
aperture and the focus of the lens 2 to execute automatic 

30 aperture control and automatic focus control, and also 
controls white balance adjustment in the camera signal 
processing section 6. 

[0045] Specifically, the central processing unit 3 con- 
trols the aperture of the lens 2 to maintain the area oc- 

35 cupied by the picked-up image of the face at a given 
luminance level, thereby executing automatic aperture 
control processing. In addition, the central processing 
unit 3 executes aperture control processing in combina- 
tion with an aperture control technique using an existing 

40 technique based on an average luminance level or the 
like measured across the entire screen, thereby making 
it possible to reliably execute aperture control even if, for 
example, an area occupied by a picked-up image of a 
face is not detected in a picked-up image of a landscape. 

45 [0046] In addition, on the basis of the size of the area 
occupied by the picked-up image of the face, the central 
processing unit 3 estimates the distance to a person hav- 
ing the face whose image is picked-up, and executes 
focus control on the basis of the distance. In this case as 

50 well, the central processing unit 3 executes focus control 
processing in combination with focus control using an 
existing technique such as a so-called hill-climbing meth- 
od of consistently performing variable control on focus in 
the direction in which the signal levels of high-frequency 

55 components increase, thereby making it possible to re- 
liably execute focus control even if an area occupied by 
a picked-up image of a face is not detected. 
[0047] In addition, the central processing unit 3 cor- 
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rects the gain of each color signal and adjusts white bal- 
ance so that the hue of the area occupied by the picked- 
up image of the face becomes equal to the hue of skin 
color having a fixed value. In this case as well, the central 
processing unit 3 executes white balance adjustment in 
combination with an existing technique, thereby making 
it possible to reliably execute white balance adjustment 
even if an area occupied by a picked-up image of a face 
is not detected. In addition, as to the combinations with 
the existing techniques, it is possible to use a wide range 
of combinations of existing techniques such as a method 
of applying an existing technique only when an area oc- 
cupied by a picked-up image of a face is not detected as 
an area larger than a given area, and a method of per- 
forming weighting addition on controlled variables includ- 
ing a controlled variable based on an existing technique 
by using a weighting coefficient based on the area or the 
like of the area occupied by the picked-up image of the 
face. 

[0048] In addition, if the user inputs an instruction to 
display the area on the basis of which the above-men- 
tioned automatic aperture control, automatic focus con- 
trol and white balance adjustment have been performed, 
the central processing unit 3 instructs the image display 
section 9 to display a frame surrounding the face. In ad- 
dition, on the basis of the settings of an operation mode 
defined by the user in advance, the central processing 
unit 3 records the detection result D1 of the face detection 
section 8 on the image recording medium 12 as image- 
annexed information D2 attached to a corresponding im- 
age file or as a separate file associated with the corre- 
sponding image file. 

[0049] In addition, if the user inputs an instruction to 
display an image file recorded on the image recording 
medium 12 for monitoring purpose, the central process- 
ing unit 3 instructs each section to cause the image dis- 
play section 9 to display the image file recorded on the 
image recording medium 12. Atthistime, if the user inputs 
an instruction to display a frame surrounding the face, 
the central processing unit 3 instructs the image display 
section 9 to display the frame surrounding the face on 
the basis of the image-annexed information D2 recorded 
on the image recording medium 12, in the case where 
the image-annexed information D2 of the image file is 
recorded on the image recording medium 12. 
[0050] In addition, if the image-annexed information 
D2 is not recorded on the image recording medium 12 
when the user inputs an instruction to display a frame 
surrounding the face, the central processing unit 3 in- 
structs the face detection section 8 to start the operation 
of processing the image data reproduced from the image 
recording medium 12 and recorded on the image RAM 
7, and acquires a detection result D1 corresponding to 
the area occupied by the picked-up image of the face. In 
addition, the central processing unit 3 instructs the image 
display section 9 to display a frame surrounding the face 
on the basis of the acquired detection result D1 . 
[0051] In addition, when the image file recorded on the 



image recording medium 12 is reproduced and the user 
inputs an instruction to perform image correction based 
on the person, the central processing unit 3 corrects the 
luminance level and the hue of the image data stored in 
5 the image RAM 7, on the basis of the area occupied by 
the picked-up image of the face, in a manner similar to 
automatic focus and white balance adjustment per- 
formed during image taking, and displays the corrected 
image date on the image display section 9. In addition, 
10 in accordance with an instruction from the user, the cen- 
tral processing unit 3 records an image file corrected for 
luminance level and hue in the above-mentioned manner 
on the image recording medium 12. The correction of 
luminance level and hue may be executed by transmitting 
15 the image data to the camera signal processing section 
6 and subjecting the image data to the processing of the 
camera signal processing section 6, or instead of using 
this process may be executed by processing the image 
data in the central processing unit 3. 
20 [0052] Accordingly, the image pickup apparatus 1 var- 
iously controls the operation of each of the image pickup 
system and the signal processing system on the basis 
of the area occupied by a picked-up image of a particular 
object, that is detected by the face detection section 8. 
25 [0053] Fig. 4 is a schematic diagram aiding in explain- 
ing the process sequence of the central processing unit 
3 responsible for the processing of detecting the area 
occupied by a picked-up image of a particular object by 
means of the face detection section 8. In the first embod- 
30 iment, the central processing unit 3 detects an area oc- 
cupied by a picked-up image of a face by sequentially 
varying on a step-by-step basis the size of an image of 
a target to be processed, with respect to a template TP 
of single size, thereby detecting at high speed an area 
35 occupied by a picked-up image of a particular object and 
appropriately setting the priority order. 
[0054] Specifically, if an image is picked-up in an im- 
age-taking mode such as the portrait mode or a self-por- 
trait mode for taking an image of the user himself/herself, 
40 a short-range view of a small number of persons tends 
to be picked-up so that the face of a larger object occupies 
a larger area in the image pickup result. Accordingly, in 
the example shown in Fig. 4, in an original image GO 
represented by original image data stored in the image 
45 RAM 7, it is determined that the main object is a person 
whose face K1 is picked-up as the largest face and per- 
sons whose faces K2 and K3 are picked-up as smaller 
faces than the face K1 are persons concerned who are 
to be given a lower priority order. 
50 [0055] Accordingly, in this case, by comparing the im- 
age of the processing target with a template while se- 
quentially varying the size of the image on a step-by-step 
basis so as to first detect the largest face K1 and then 
sequentially detect the faces K2 and K3 picked-up as 
55 smaller faces than the face K1 , it is possible to sequen- 
tially detect areas occupied by the respective faces, in 
orderfrom the main person. In addition, in this case, even 
if the process is stopped at a given stage after the face 
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of the main person has been detected, it is possible to 
detect areas respectively occupied by picked-up images 
of faces within a practically sufficient accuracy for the 
processing to be performed in the image pickup appara- 
tus 1. 

[0056] Specifically, in the example shown in Fig. 4, the 
resolution of the original image GO represented by the 
original image data is sequentially reduced on a step-by- 
step basis to generate reduced images G7, G6, G5, 
G1 , and the areas occupied by the picked-up images of 
the respective faces are detected in each of the reduced 
images G7, G6, G5, G1 and the original image GO by 
template matching using the template TP, so that the 
faces K1 and K2 are respectively detected in the reduced 
images G6 and G4 and the face K3 is detected in the 
original image GO. Accordingly, in this case, in order that 
the areas be sequentially detected in order from the larg- 
est picked-up face, as indicated by an arrow A, the areas 
occupied by the picked-up images of the respective faces 
are first detected in the reduced image G7 representing 
the original image on the most reduced scale, and then 
the areas occupied by the picked-up images of the re- 
spective faces in each of the other images are sequen- 
tially detected while the sizes of the respective images 
to be processed are being sequentially enlarged on a 
step-by-step basis toward the size of the original image 
GO. In this manner, the areas occupied by the picked-up 
images of the respective faces can be detected in the 
priority order of detection. 

[0057] As opposed to this processing, if picked-up im- 
ages of a larger number of persons are contained in a 
group photograph, the larger the number of picked-up 
persons, unnecessary template matching will be execut- 
ed on images reduced to the greater extent. For this rea- 
son, in this case, as indicated by an arrow B, the areas 
occupied by the picked-up images of the respective faces 
are sequentially detected while the images to be proc- 
essed are being sequentially reduced on a step-by-step 
basis, so that the areas occupied by the picked-up im- 
ages of the respective faces can be detected in a short 
time after the start of the processing. 
[0058] In addition, there is a case where a face is not 
contained in an image picked-up in a landscape mode 
or the like, and even if a face is contained in the image, 
the area occupied by a picked-up image of the face is 
extremely small in many cases such as a distant view of 
a person against the background of a landscape and a 
picked-up image of the face of a passerby. In such a case 
as well, it is estimated that the presence of a large face 
which occupies the entire screen is a very rare case. 
Accordingly, in this case, faces are sequentially detected 
on a step-by-step basis in order from the smallest to the 
largest areas, and even if the detection processing of the 
areas occupied by the picked-up images of the respective 
faces is stopped during the sequence of the processing, 
the areas occupied by the picked-up images of the re- 
spective faces can be detected with practically sufficient 
accuracy. 



[0059] In this manner, the central processing unit 3 ex- 
ecutes the process sequence shown in Figs. 1 and 5 and 
detects the areas occupied by the picked-up images of 
the respective faces while sequentially varying the size 
5 of the target image on a step-by-step basis. Referring to 
Fig. 1 , when the central processing unit 3 starts the proc- 
ess sequence, the process proceeds from step SP1 to 
step SP2, in which the central processing unit 3 gener- 
ates image data representative of the original image GO 
10 and stores the image data in the image RAM 7 under the 
control of the camera signal processing section 6. In step 
SP3, the central processing unit3 controls the operations 
of the resolution conversion section 8A and the image 
memory 8C by means of the controller 8B provided in 
15 the face detection section 8, and sequentially switches 
the scale factor a as shown in Fig. 6 to convert the res- 
olution of the image data representative of the original 
image GO stored in the image RAM 7 and store the con- 
verted image data in the image RAM 7, thereby storing 
image data representative of the reduced images G1 to 
G7 in the image RAM 7. In this case, as indicated by the 
arrow B in Fig. 4, the reduced images may be sequentially 
reduced at a given scale factor to generate a plurality of 
reduced images G1 to G7 having different resolutions. 
[0060] Then, in step SP4, the central processing unit 
3 determines whether the areas occupied by the picked- 
up images of the respective faces are to be detected in 
order from the largest face or in order from the smallest 
face. The central processing unit 3 executes the decision 
processing of step SP4 on the basis of the image-taking 
mode and on the basis of a result of processing of so-far 
continuing frames which are the past detection result of 
the face detection core 8D. Specifically, if the image-tak- 
ing mode of the image pickup apparatus 1 is set to an 
image-taking mode, such as the portrait mode or the self- 
portrait mode, which can be predicted to be used to take 
an image of a short-range view of a small number of 
persons, the central processing unit 3 determines so as 
to detect the areas occupied by the picked-up images of 
the respective faces in order from the largest face. In 
addition, even if the central processing unit 3 cannot de- 
termine owing to such an image-taking mode, if an image 
in which the faces of one to several persons are picked- 
up with comparatively large areas continues by several 
frames, the central processing unit 3 determines so as 
to detect the areas occupied by the picked-up images of 
the respective faces in order from the largest face. 
[0061 ] As opposed to this processing, if the image-tak- 
ing mode of the image pickup apparatus 1 is set to an 
image-taking mode, such as the landscape mode, which 
can be predicted to be used to take an image of a multi- 
plicity of persons having comparatively small areas, the 
central processing unit 3 determines so as to detect the 
areas occupied by the picked-up images of the respective 
faces in order from the smallest face. In addition, even if 
the central processing unit 3 cannot determine owing to 
such an image-taking mode, if an image in which a mul- 
tiplicity of faces are picked-up with comparatively small 
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areas continues by several frames, the central process- 
ing unit 3 determines so as to detect the areas occupied 
by the picked-up images of the respective faces in order 
from the smallest face. 

[0062] If the central processing unit3 cannotdetermine 
in step SP4 in which order to detect the areas occupied 
by the picked-up images of the respective faces, the cen- 
tral processing unit 3 performs face detection in either of 
the orders in accordance with initial settings, and in the 
first embodiment, the initial settings are set to face de- 
tection to be performed in order from the largest face. 
These initial settings may also use the history of use of 
the image pickup apparatus 1 , the settings of the user, 
and the like. 

[0063] If the central processing unit 3 determines in 
step SP4 that the areas occupied by the picked-up im- 
ages of the respective faces are to be detected in order 
from the largest order, the process proceeds from step 
SP4 to step SP5, in which the central processing unit 3 
loads the image data representative of the reduced im- 
age G7 stored in the image RAM 7 into the image memory 
8C and transmits the loaded image data to the face de- 
tection core 8D to set the smallest reduced image G7 to 
a target image to be processed, and the face detection 
core 8D detects the areas occupied by the picked-up 
images of the respective faces by template matching. 
Thus, in the example shown in Fig. 4, the areas occupied 
by the picked-up images of the respective faces are not 
detected in the reduced image G7. 
[0064] Then, in step SP6, the central processing unit 
3 sets the reduced image G6 one step larger than the 
reduced image G7 to a target image to be processed and 
executes similar processing, and detects the face K1 in 
the example shown in Fig. 4. In addition, the central 
processing unit 3 sequentially repeats this processing to 
detect the areas occupied by the picked-up images of 
the respective faces by template matching using the re- 
duced images G1 to G7 and the original image GO. In 
addition, in this processing, the central processing unit 3 
designates the order of scanning, the start position of 
scanning and the like in the face detection core 8D ac- 
cording to the image-taking mode. 
[0065] The central processing unit 3 sequentially exe- 
cutes the processing of face detection on a step-by-step 
basis by using the reduced images G1 to G7 and the 
original image GO, and determines whether the process 
is to be stopped at a given processing stage. Accordingly, 
in the processing shown in Fig. 1, when the central 
processing unit 3 processes the reduced image G6 in 
step SP7, the central processing unit 3 determines in 
step SP8 whether the process is to be stopped. In addi- 
tion, the processing of determining whether to stop the 
process may be executed at each stage, and may also 
be executed at a stage based on settings or the like de- 
fined by the user. 

[0066] In step SP8, the central processing unit 3 de- 
termines whether the process is to be stopped according 
to the past detection result and the image-taking mode. 



Specifically, if an area occupied by a picked-up image of 
a face, which area corresponds to a frame surrounding 
the face which is to be displayed on the image display 
section 9, is already detected in the processing that has 

5 so far been performed, the central processing unit 3 de- 
termines that the process is to be stopped. In addition, if 
the image-taking mode is set to, for example, the portrait 
mode or the self-portrait mode, the central processing 
unit 3 determines that the process is to be stopped at a 

10 stage corresponding to the set image-taking mode. In 
addition, the central processing unit 3 may also deter- 
mine whether to stop the process, on the basis of settings 
defined by the user. 

[0067] Accordingly, the central processing unit 3 pro- 
fs ceeds from step SP8 to step SP9 and stops the process- 
ing of template matching using the remaining reduced 
images G3 to G1 and the original image GO. Then, the 
central processing unit 3 switches a reduced image to 
be subjected to the processing of template matching, on 
20 the basis of the past detection result and in accordance 
with the image-taking mode of an image pickup section. 
In addition, during the switching of the target image ac- 
cording to the image-taking mode, the switching of a re- 
duced image at which to start the process may be exe- 
25 cuted instead of or in addition to the switching of a re- 
duced image at which to end the process. 
[0068] The central processing unit3 determines in step 
SP9whetheran imageof thesubsequentframe has been 
acquired, and if it is determined in step SP9 that an image 
30 of the subsequent frame has been acquired, the central 
processing unit 3 returns to step SP2 and repeats the 
processing. If it is determined in step SP9 that an image 
of the subsequent frame has not yet been acquired, the 
central processing unit 3 proceeds from step SP9 to step 
35 SP1 0 and ends the process sequence. 

[0069] Conversely, if the result is negative in step SP8, 
the central processing unit 3 sequentially executes the 
processing of template matching by using the remaining 
reduced images G3 to G1 and the original image GO, and 
40 proceeds to step SP9. 

[0070] As opposed to this processing, if the central 
processing unit 3 determines in step SP4 that the areas 
occupied by the picked-up images of the respective faces 
are to be detected in order from the smallest order, the 
45 process proceeds from step SP4 to step SP1 3, in which 
the central processing unit 3 loads the image data rep- 
resentative of the original image GO stored in the image 
RAM 7 into the image memory 8C and transmits the load- 
ed image data to the face detection core 8D to set the 
50 largest original image GO to a target image to be proc- 
essed, and the face detection core 8D detects the areas 
occupied by the picked-up images of the respective faces 
by template matching. Thus, in the example shown in 
Fig. 4, the smallest face K3 is detected in the original 
55 image GO. 

[0071] Then, in step SP14, the central processing unit 
3 sets the reduced image G1 one step smaller than the 
original image GO to a target image to be processed and 



9 



17 



EP 1 785 914 A1 



18 



executes similar processing, and sequentially repeats 
this processing to detect the areas occupied by the 
picked-up images of the respective faces by template 
matching using the reduced images G1 to G7 and the 
original image GO in order from the smallest image. In 
addition, in this processing as well, the central processing 
unit 3 designates the order of scanning, the start position 
of scanning and the like in the face detection core 8D 
according to the image-taking mode and the like. 
[0072] The central processing unit 3 sequentially exe- 
cutes the processing of face detection on a step-by-step 
basis in order from the smallest image by using the re- 
duced images G1 to G7 and the original image GO, and 
determines whether the process is to be stopped at a 
given processing stage. Accordingly, in the processing 
shown in Fig. 5, when the central processing unit 3 proc- 
esses the reduced image G4 in step SP15, the central 
processing unit 3 determines in step SP16 whether the 
process is to be stopped. In addition, the processing of 
determining whether to stop the process may be execut- 
ed at each stage, and may also be executed at a stage 
based on settings or the like defined by the user. 
[0073] In step SP15, the central processing unit 3 de- 
termines whether the process is to be stopped according 
to the past detection result. Specifically, for example, if 
an area occupied by a picked-up image of a face cannot 
be detected during several successive steps after an ar- 
ea occupied by a picked-up image of a face has been 
detected at the past stage, and furthermore if an area 
occupied by an image of a face picked-up at a later stage 
is not yet detected during the past successive frames, 
the central processing unit 3 determines that the process 
is to be stopped. In addition, the central processing unit 
3 may be adapted to stop the process after a given stage 
during the landscape mode, for example. Accordingly, 
the central processing unit 3 may be adapted to deter- 
mine whether to stop the process, according to the im- 
age-taking mode, and may also be adapted to determine 
whether to stop the process, on the basis of settings de- 
fined by the user. 

[0074] Accordingly, in this case, the central processing 
unit 3 proceeds from step SP16 to step SP9 and stops 
the processing of template matching using the remaining 
reduced images G5 to G7. In addition, in step SP9, the 
central processing unit 3 determines whether an image 
of the subsequent frame has been acquired, and if it is 
determined in step SP9 that an image of the subsequent 
frame has been acquired, the central processing unit 3 
returns to step SP2 and repeats the processing. Con- 
versely, if it is determined in step SP9 that an image of 
the subsequent frame has not yet been acquired, the 
central processing unit 3 proceeds from step SP9 to step 
SP1 0 and ends the process. Accordingly, in this case as 
well, the central processing unit 3 switches a reduced 
image to be subjected to the processing of template 
matching, on the basis of the past detection result and 
furthermore in accordance with the image-taking mode 
of the image pickup section. I n this case as well, the target 



image to be processed may be switched by the switching 
of a reduced image at which to start the process, instead 
of or in addition to the switching of a reduced image at 
which to end the process. 
5 [0075] Conversely, if the result is negative in step 
SP1 6, the central processing unit3 sequentially executes 
the processing of template matching by using the remain- 
ing reduced images G5 to G7, and proceeds to step SP9. 
[0076] In addition, instead of preparing in advance a 
10 plurality of kinds of reduced images having different sizes 
and storing them in the image RAM 7 so that the process- 
ing of template matching is executed at each stage, it is 
also preferable to adopt a construction in which each time 
the processing of template matching is to be executed 
at an individual stage, a reduced image having a corre- 
sponding size is prepared. According to this construction, 
it is possible to omit the processing of preparing unnec- 
essary reduced images, and furthermore, since it is pos- 
sible to omit the processing of writing image data related 
to reduced images into the image RAM 7, the time re- 
quired for the processing can be reduced by that amount. 
In addition, in this case, instead of reducing an original 
image, it is also possible to reduce a reduced image used 
at the preceding stage and generate a reduced image to 
be processed at the succeeding stage. 
[0077] In this manner, the central processing unit 3 se- 
quentially detects on a step-by-step basis the areas oc- 
cupied by the picked-up images of the respective faces 
and acquires the face detection result D1 from the face 
detection core 8D on the basis of position information on 
the areas occupied by the picked-up images of the re- 
spective faces. 

[0078] As shown in Fig. 7, the central processing unit 
3 divides the size of the template TP by the scale factor 
a of the reduced image G6 subjected to the face detection 
and enlarges the size of the template TP by the scale 
factor a, and calculates the area occupied by the picked- 
up image of the face on the original image GO from the 
enlarged size and position information corresponding to 
the face detection result D1 . Then, the central processing 
unit 3 corrects the face detection result D1 using the tem- 
plate TP of single size. In addition, on the basis of the 
corrected face detection result D1 , the central processing 
unit 3 instructs the image display section 9 to display a 
frame surrounding the face. The central processing unit 
3 also generates the image-annexed information D2 on 
the basis of the corrected face detection result D1 . 
[0079] In addition, on the face detection result D1 cor- 
rected in this manner, the central processing unit 3 ac- 
quires image data representative of the area occupied 
by the picked-up image of the face and executes the 
processing of aperture control, focus control and white 
balance adjustment. At this time, the central processing 
unit 3 sets the priority order on the basis of the detection 
order, and executes the processing of aperture control, 
focus control and white balance adjustment on the basis 
of the largest picked-up face when the areas occupied 
by the picked-up images of the respective faces are de- 
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tected in accordance with this priority order, for example, 
in order from the largest face. At this time, if an area 
occupied by a picked-up image of another face is also 
detected, the central processing unit 3 executes the 
processing of aperture control, focus control and white 
balance adjustment with reference to the luminance lev- 
el, the size and the like of that area. The processing in- 
cludes, for example, the processing of performing weight- 
ing addition on controlled variables found from the re- 
spective areas to calculate a final controlled variable, and 
executing processing such as aperture control on the ba- 
sis of the final controlled variable. 

(2) Operation of the First Embodiment 

[0080] In the image pickup apparatus 1 (Fig. 2) having 
the above-mentioned construction, an image pickup re- 
sult is acquired by the image sensor 4 in the form of a 
moving image or a still image and is variously corrected 
by the camera signal processing section 6, and the cor- 
rected image pickup result is stored in the image RAM 
7. The image pickup result stored in the image RAM 7 is 
subjected to monitoring on the image display section 9, 
and is compressed by the image compression/decom- 
pression section 1 0 and recorded on the image recording 
medium 12, in accordance with an instruction from the 
user. Otherwise, the image pickup result may be output- 
ted to external devices via an input/output terminal. 
[0081] During this processing, in the image pickup ap- 
paratus 1, monitoring image data is generated by reso- 
lution conversion processing in the camera signal 
processing section 6, and the monitoring image data is 
used to monitor the image pickup result on the image 
display section 9. The monitoring image data is also set 
to image data representative of an original image to be 
subjected to face detection, and is inputted to the reso- 
lution conversion section 8A of the face detection section 
8 (Fig. 3), in which the resolution of the input image data 
is reduced by a predetermined value to generate image 
data representative of a reduced image. This image data 
representative of the reduced image is stored into the 
image RAM 7 via the image memory 8C. In the image 
pickup apparatus 1 , the processing by the resolution con- 
version section 8A and the image memory 8C is repeat- 
ed, so that image data representative of a plurality of 
reduced images which have been sequentially reduced 
in resolution with respect to the original image on a step- 
by-step basis are stored in the image RAM 7. 
[0082] In the image pickup apparatus 1, image data 
representative of a plurality of reduced images are gen- 
erated in this manner in advance and stored in the image 
RAM 7, and the image data are sequentially transmitted 
to the image core 8D. The image data are processed by 
template matching using a template, so that an area oc- 
cupied by a picked-up image of a particular object cor- 
responding to the template is detected in the reduced 
image. In the image pickup apparatus 1, the particular 
object is set to the face of a person, and in the image 



core 8D the plurality of reduced images are processed 
in an order in which their resolution varies sequentially 
on a step-by-step basis. Accordingly, the areas occupied 
by the picked-up images of the respective particular ob- 

5 jects are detected in order from the largest object or in 
order from the smallest object, so that the priority order 
can be set on the basis of the order of detection. In ad- 
dition, the process can be stopped as needed to detect 
an objective area in a short time, so that the areas occu- 

10 pied by the picked-up images of the respective faces can 
be detected at high speed to appropriately set the priority 
order. 

[0083] In addition, since the reduced images are proc- 
essed by template matching while the resolution is being 

15 sequentially varied on a step-by-step basis in the above- 
mentioned manner, the areas of various sizes occupied 
by the picked-up images of the respective faces can be 
detected by using a template of single size, so that the 
construction of the image pickup apparatus 1 can be 

20 made simple by that amount. 

[0084] In the image pickup apparatus 1 , the image data 
related to targets to be processed are image data ac- 
quired by the image sensor 4, which serves as an image 
pickup section, and the camera signal processing section 

25 6, and aperture control, focus control and white balance 
adjustment are executed on the basis of the areas occu- 
pied by the picked-up images of the respective faces de- 
tected in this manner, so that the areas occupied by the 
picked-up images of the respective particular objects can 

30 be detected at high speed to appropriately set the priority 
order. Accordingly, while the processing of aperture con- 
trol, focus control and white balance adjustment is being 
accurately executed on an object desired by the user, 
individual images of successive frames can be proc- 

35 essed in a short time and this processing can be reliably 
executed. 

[0085] In addition, image data acquired from the image 
recording medium 12 and image data acquired from the 
data communication section can be displayed on the im- 

40 age display section 9 in the same manner as targets to 
be processed, and furthermore, these image data can 
be variously corrected and reliably processed. 
[0086] In addition, the order of processing of the re- 
duced images is switched according to the image-taking 

45 mode and the past detection result, so that unnecessary 
processing can be reduced and an area occupied by a 
picked-up image of a predetermined object can be de- 
tected in a short time. 

[0087] Specifically, if it is estimated that a short-range 
50 view of persons has been picked-up, reduced images 
having different resolutions are sequentially processed 
on a step-by-step basis to detect their faces in order from 
largest, so that the area occupied by a picked-up image 
of a face which is a desired object can be detected in a 
55 short time. Conversely, if it is estimated that a group pho- 
tograph or the like of persons has been picked-up, re- 
duced images having different resolutions are sequen- 
tially processed on a step-by-step basis to detect their 
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faces in order from smallest, so that the area occupied 
by a picked-up image of a face which is a desired object 
can be detected in a short time. 

[0088] In addition, in the case where the reduced im- 
ages are sequentially processed in this manner, the 
processing of unnecessary reduced images can be omit- 
ted by stopping the process halfway according to the im- 
age-taking mode and the past detection result and fur- 
thermore changing the start of the process to switch re- 
duced images to be subjected to the process. Accord- 
ingly, it is possible to detect in a short time the area oc- 
cupied by a face which is a desired object. 
[0089] Accordingly, in the image pickup apparatus 1, 
the area occupied by a picked-up image of a face which 
is a desired object is detected in each of the reduced 
images, and the size of the template is converted to the 
size on the original image according to the size of each 
reduced image with respect to the original image and the 
area occupied by the picked-up image of the particular 
object is detected on the original image. In addition, 
processing such as aperture control is executed accord- 
ing to the size detected on the original image. 
[0090] During this processing, in the image pickup ap- 
paratus 1 , the priority order is set in accordance with the 
order of detection and the detection result of face detec- 
tion is processed, so that processing such as focus con- 
trol can be reliably executed by simple processing. 

(3) Advantage of the First Embodiment 

[0091] According to the above-mentioned construc- 
tion, reduced images are sequentially subjected to tem- 
plate matching in an order in which their resolution varies 
sequentially on a step-by-step basis, thereby detecting 
the areas respectively occupied by picked-up images of 
particular objects. Accordingly, the areas respectively oc- 
cupied by picked-up images of particular objects can be 
detected at high speed and the priority order can be ap- 
propriately set. 

[0092] In addition, the image acquisition section that 
acquires target images to be processed is an image pick- 
up section, so that if the first embodiment is applied to 
image pickup equipment, the usefulness of image pickup 
apparatus can be improved by effectively using detection 
results in focus control, aperture control, white balance 
adjustment and the like. 

[0093] In addition, since the image acquisition section 
is an image pickup section related to recording media or 
has a construction related to data communication with 
external devices, the usefulness of image pickup appa- 
ratus can be improved by effectively using detection re- 
sults in monitoring and the like of an image pickup result 
recorded on a recording medium. 

[0094] At this time, the reduced images are processed 
in an order in which their resolution sequentially increas- 
es or in an order in which their resolution sequentially 
decreases, so that when an image of a short-range view 
or a distant view of a desired object is picked-up, the area 



occupied by the picked-up image of the desired object 
can be detected in a short time. 

[0095] As mentioned above, an area occupied by a 
picked-up image of a face which is a desired object is 

5 detected in each reduced image, and the size of the tem- 
plate is converted to the size on the original image ac- 
cording to the size of each reduced image with respect 
to the original image and an area occupied by a picked- 
up image of a particular object is detected on the original 

10 image. Accordingly, various control and processing can 
be executed on the basis of the area occupied by the 
picked-up image of the particular object, which area is 
detected on the original image. 

[0096] In addition, areas respectively occupied by 
15 picked-up images of particular objects are detected in 
each of reduced images whose resolution is set sequen- 
tially on a step-by-step basis, and the priority order of the 
areas occupied by the picked-up images of the particular 
objects is set in accordance with the order of detection. 
20 Accordingly, the priority order can be set simply and high- 
ly accurately, so that control and processing can be var- 
iously executed on the basis of the priority order. 
[0097] In addition, the order of processing of a plurality 
of reduced images is switched on the basis of the past 
25 detection result and furthermore in accordance with the 
image-taking mode, so that an area occupied by a picked- 
up image of a particular object can be detected in a short 
time after the start of processing. 

[0098] In addition, reduced images to be processed 
30 are switched on the basis of the past detection result and 
furthermore in accordance with the image-taking mode, 
so that unnecessary processing of reduced images can 
be omitted to reduce the time required for processing to 
a further extent. 
35 [0099] A second embodiment of the present invention 
will be described below. 

[01 00] Fig. 8 is a schematic diagram aiding in explain- 
ing, in comparison with Figs. 4 and 6, processing to be 
executed by an image pickup apparatus according to the 

40 second embodiment of the present invention. In the sec- 
ond embodiment, the reduced image G4 to be used as 
a reference for intermediate processing is generated and 
the reduced images G5 to G7 lower in resolution than 
the reduced image G4 are generated by converting the 

45 resolution of the reference reduced image G4, instead 
of by the resolution conversion processing of an original 
image in which the scale factor is sequentially switched 
on a step-by-step basis or instead of by the sequential 
step-by-step resolution conversion processing of an orig- 

50 inal image in which the scale factor is maintained at a 
constant value. In addition, the reduced images G1 to 
G3 larger in resolution than the reduced image G4 are 
generated by converting the resolution of the original im- 
age GO. 

55 [0101] According to the second embodiment, a re- 
duced image to be used as a reference for intermediate 
processing is generated and reduced images are gener- 
ated from the reference reduced image and an original 
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image, so that the access efficiency of a memory which 
records and holds these reduced images is improved 
and, furthermore, the address control of the memory can 
be made simple. 

[0102] A third embodiment of the present invention will 
be described below. 

[0103] Fig. 9 is a block diagram showing a recording 
and reproducing apparatus according to the third embod- 
iment of the present invention. A recording and repro- 
ducing apparatus 21 is, for example, a DVD (Digital Ver- 
satile Disk) recorder which processes video content ob- 
tained from a tuner, which is not shown, by an image 
processing section 22 and then records the processed 
video content on the recording medium 12 using an op- 
tical disk. In the recording and reproducing apparatus 21 , 
identical reference numerals are used to denote the 
same constituent elements as the corresponding ones 
used in the image pickup apparatus 1 mentioned above 
in connection with the first embodiment, and repetition 
of the same description is omitted. 
[0104] A central processing unit 23 controls the oper- 
ations of the respective sections to control the operation 
of the recording and reproducing apparatus 21 , by exe- 
cuting a processing program recorded on a memory 14. 
In this sequence of control, if a user gives an instruction 
to detect a particular object, an area occupied by a 
picked-up image of the particular object is detected by 
the face detection section 8 and the detection result is 
displayed on the image display section 9. Incidentally, 
the particular object is, for example, a favorite actor of 
the user. 

[01 05] Even in the case of the third embodiment which 
is applied to process various video contents in a recording 
and reproducing apparatus, it is possible to obtain an 
advantage similar to that of the first embodiment. 
[0106] A fourth embodiment of the present invention 
will be described below. 

[01 07] I n the above description of the first embodiment, 
reference has been made to the case where aperture 
control is performed so that a luminance level is main- 
tained at a predetermined value in an area occupied by 
a picked-up image of a face of the highest priority order, 
as well as to the case where the processing of aperture 
control is executed with a final control variable based on 
weighting addition of the area and an area occupied by 
a picked-up image of another face. However, the present 
invention is not limited to either of these cases, and the 
aperture and the charge storage time of an image pickup 
apparatus are controlled so that the luminance level is 
maintained at a predetermined value and, at the same 
time, the depth of field is controlled. In this case, even if 
the distances to various objects differ from one another, 
the depth of field can be set to prevent defocusing within 
the range of these different distances, thereby improving 
the usefulness of the apparatus to a further extent. 
[0108] The above description of any of the first to fourth 
embodiments has referred to the case where an area 
occupied by a picked-up image of a face is detected, but 



the present invention is not limited to such a case and 
can be widely applied to various cases where various 
templates are applied to detect areas occupied by 
picked-up images of various objects, for example, a case 
5 where an area occupied by a picked-up image of a user's 
child needs to be detected. 

[0109] In the above description of the first to fourth em- 
bodiments, reference has been made to the cases where 
the present invention is applied to an image pickup ap- 
10 paratus and a recording and reproducing apparatus us- 
ing an optical disk, but the present invention is not limited 
to such a case and can be widely applied to recording 
and reproducing equipment using various recording me- 
dia as well as image processing equipment such as print- 
's ers. The present invention can be further applied to image 
processing software for computers and the like. 
[0110] The present invention contains subject mater 
related to Japanese Patent Application No. 
JP2005-328256 filed in the Japanese Patent Office on 
20 November 14, 2005, the entire contents of which being 
incorporated herein by reference. 



Claims 

25 

1. An image processing apparatus comprising: 

an image acquisition section which acquires an 
original image; 

30 a resolution conversion section which converts 

the resolution of said original image acquired by 
said image acquisition section and generates a 
plurality of reduced images having different res- 
olutions; 

35 a detection section which processes by template 

matching using a template said plurality of re- 
duced images generated by said resolution con- 
version section and detects an area occupied 
by a picked-up image of a particular object cor- 

40 responding to said template, from said reduced 

images; and 

a detection result processing section which de- 
tects said area occupied by said picked-up im- 
age of said particular object on said original im- 

45 age, by processing a detection result obtained 

by said detection section, 
said detection section detecting said area occu- 
pied by said picked-up image of said particular 
object by processing said plurality of reduced 

50 images in an order in which resolution sequen- 

tially varies on a step-by-step basis. 

2. The image processing apparatus according to claim 
1, 

55 wherein said image acquisition section is an image 
pickup section which acquires image data related to 
said original image based on an image pickup result, 
and 
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wherein said image processing apparatus has a re- 
cording medium which records said image data ac- 
quired by said image pickup section. 

3. The image processing apparatus according to claim 
1, 

wherein said image acquisition section is an image 
acquisition section associated with a recording me- 
dium, for acquiring image data related to said original 
image recorded on said recording medium. 

4. The image processing apparatus according to claim 
1, 

wherein said image acquisition section is adata com- 
munication section which performs data communi- 
cation with an external device. 

5. The image processing apparatus according to claim 
1 ,furthercomprising a storage section which records 
and holds said plurality of reduced images generated 
by said resolution conversion section, 

wherein said detection section sequentially process- 
es said plurality of reduced images recorded on said 
storage section and processes said plurality of re- 
duced images generated by said resolution conver- 
sion section. 

6. The image processing apparatus according to claim 
1 , wherein said order related to said processing by 
said detection section, in which said resolution se- 
quentially varies on a step-by-step basis, is an order 
in which said resolution sequentially increases. 

7. The image processing apparatus according to claim 
1, wherein said order related to said processing by 
said detection section, in which said resolution se- 
quentially varies on a step-by-step basis, is an order 
in which said resolution sequentially decreases. 

8. The image processing apparatus according to claim 
1, wherein said detection result processing section 
converts the size of said template to a size on said 
original image on the basis of the size of said reduced 
image with respect to said original image 

9. The image processing apparatus according to claim 
1, wherein said detection result processing section 
sets the priority order of said area occupied by said 
picked-up image of said particular object on said 
original image, on the basis of the order of detection 
detected by said detection section. 

10. The image processing apparatus according to claim 
1 , further comprising a control section which controls 
the operation of each of said sections, 

wherein said control section switches the order of 
processing of said plurality of reduced images relat- 
ed to processing in said detection section, on the 



basis of a past detection result obtained by said de- 
tection section. 

1 1 . The image processing apparatus according to claim 
5 2, further comprising a control section which controls 

the operation of each of said sections, 
wherein said control section switches the order of 
processing of said plurality of reduced images relat- 
ed to processing in said detection section, in accord- 
10 ance with an image-taking mode which is set in said 
image pickup section. 

12. The image processing apparatus according to claim 

1 , further comprising a control section which controls 
15 the operation of each of said sections, 

wherein said control section switches said reduced 
images subjected to processing in said detection 
section on the basis of a past detection result ob- 
tained by said detection section. 

20 

13. The image processing apparatus according to claim 

2, further comprising a control section which controls 
the operation of each of said sections, 

wherein said control section switches said reduced 
25 images subjected to processing in said detection 
section, in accordance with an image-taking mode 
which is set in said image pickup section. 

14. An image processing method comprising: 

30 

an image acquisition step of acquiring an original 
image; 

a resolution conversion step of converting the 
resolution of said original image acquired in said 
35 image acquisition step and generating a plurality 

of reduced images having different resolutions; 
a detection step of processing by template 
matching using a template said plurality of re- 
duced images generated in said resolution con- 
40 version step and detecting an area occupied by 

a picked-up image of a particular object corre- 
sponding to said template, from said reduced 
images; and 

a detection result processing step of detecting 
45 said area occupied by said picked-up image of 

said particular object on said original image, by 
processing a detection result obtained in said 
detection step, 

said detection step detecting said area occupied 
50 by said picked-up image of said particular object 

by processing said plurality of reduced images 
in an order in which resolution sequentially var- 
ies on a step-by-step basis. 

55 15. A program for an image processing method, which 
processes images by being executed by operation 
processing means, comprising: 
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an image acquisition step of acquiring an original 
image; 

a resolution conversion step of converting the 
resolution of said original image acquired in said 
image acquisition step and generating a plurality 5 
of reduced images having different resolutions; 
a detection step of processing by template 
matching using a template said plurality of re- 
duced images generated in said resolution con- 
version step and detecting an area occupied by 10 
a picked-up image of a particular object corre- 
sponding to said template, from said reduced 
images; and 

a detection result processing step of detecting 
said area occupied by said picked-up image of 15 
said particular object on said original image, by 
processing a detection result obtained in said 
detection step, 

said detection step detecting said area occupied 
by said picked-up image of said particular object 20 
by processing said plurality of reduced images 
in an order in which resolution sequentially var- 
ies on a step-by-step basis. 

16. A recording medium which records a program for an 25 
image processing method of processing images by 
being executed by operation processing means, said 
program comprising: 

an image acquisition step of acquiring an original 30 
image; 

a resolution conversion step of converting the 
resolution of said original image acquired in said 
image acquisition step and generating a plurality 
of reduced images having different resolutions; 35 
a detection step of processing by template 
matching using a template said plurality of re- 
duced images generated in said resolution con- 
version step and detecting an area occupied by 
a picked-up image of a particular object corre- *o 
sponding to said template, from said reduced 
images; and 

a detection result processing step of detecting 
said area occupied by said picked-up image of 
said particular object on said original image, by 45 
processing a detection result obtained in said 
detection step, 

said detection step detecting said area occupied 
by said picked-up image of said particular object 
by processing said plurality of reduced images 50 
in an order in which resolution sequentially var- 
ies on a step-by-step basis. 
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